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Abstract 

In this paper we consider the problem of minimizing a quadratic functional for a discrete- 
time linear stochastic system with multiplicative noise, on a standard probability space, in 
infinite time horizon. We show that the necessary and sufficient conditions for the existence 
of the optimal control can be formulated as matrix inequalities in frequency domain. Further- 
more, we show that if the optimal control exists, then certain Lyapunov equations must have 
a solution. The optimal control is obtained by solving a deterministic linear-quadratic opti- 
mal control problem whose functional depends on the solution to the Lyapunov equations. 
Moreover, we show that under certain conditions, solvability of the Lyapunov equations is 
guaranteed. We also show that, if the frequency inequalities are strict, then the solution is 
unique up to equivalence. 
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1 Introduction 



Kalman- Yakubovich Lemma (KY Lemma) was a groundbreaking result that paved way for a 
solutions to lots of problems in control theory, including optimal control. The first variant of 
the Lemma was derived by Yakubovich in 1962 (see [22] ). The following year, the discrete-time 
version of that result was derived by Szego and Kalman (see [20]). It is called sometimes the 
Kalman-Szego Lemma (KS Lemma); see [5].l 1 3 1.IT7] for a comprehensive review of various results 
in control theory derived from the KY Lemma. Various works such as [IH1] considered problems 
with quadratic functionals whereas Yakubovich (see [25 , 26J ) derived the KY Lemma for the case 
in which both the control and state vectors are both Hilbert spaces. 

Dokuchaev [6] considered a continuous time stochastic linear-quadratic optimal control prob- 
lem, with the state evolution described by Ito equations, with state dependent coefficients; a 
generalization of the Frequency Theorem was obtained. We consider a discrete-time analogy 
of the problem studied in [6]. We show that the necessary and sufficient conditions for the 
existence of the optimal control can be formulated as matrix inequalities in frequency domain. 
Furthermore, we show that if the optimal control exists, then certain Lyapunov equations must 
have a solution. The optimal control is obtained by solving a deterministic linear-quadratic 
optimal control problem whose functional depends on the solution to the Lyapunov equations. 
Moreover, we show that under certain conditions, solvability of the Lyapunov equations is guar- 
anteed. We also show that, if the frequency inequalities are strict, then the solution is unique 
up to equivalence. 

2 Problem Statement 

We consider the following optimization problem on a standard probability space, (O, JF, P). 




(2.1) 



t=o 



over the set 



U 




(2.2) 



subject to 



x t+1 = Ax t + bu t + Cxtit+i 



£ = 0,1,2,... 



(2.3) 



xq = a. 



(2.4) 
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Here xt is a random ra-vector of states, ut is an m-vector of controls and U is the set of 
admissible controls. Matrices A G R nXT \ b G R nxm , C G R nxr \ G = G T G R nxn , 7 G iT x?1 , 
and r = r T G m x m are constant. The scalar ^ G R is the discrete-time white noise adapted 
to a flow of non-decreasing cr-algebras ^jc^ such that E£t = 0, Var = 1. The vector a is 
random, measurable with respect to J^o> independent of {^t}^ 1 and is such that E \a\ 2 < +00 
and E |aa T | 2 < +00; we denote by |.| the Euclidean norm for vectors and Frobenius norm for 
matrices. 

We assume all the matrices in (|2.2[) and (|2.3j) are real and we restrict our considerations 
to the case when all eigenvalues A (A) of A lie inside the unit disk on the complex plane (that 
is, the spectral radius of A is p(A) < 1). Moreover, we assume that the system is stable 
in mean-square sense for Ut = 0. Various sufficient conditions of this stability can be found 
in [7HI2CE3CE71CE81EI] and other works. 

+00 

For random xt, yt G C n we denote the inner product (x.,y.) by (x.,y.) = Yl ^ x JVt &n< ^ tlie- 

t=0 

+00 

norm by = W (x,,x,). Furthermore, we write \\x.\\i = \ x t\i where \x\i is the /i-norm 

t=o 

\x\i = of a vector x or an entrywise Zi-norm \x\\ = Y^ij \ x ij\ of a matrix x. 

3 Main Results 

Condition 3.1. There exist symmetric matrices H and in C nxn satisfying 

A 1 HA- H + 9 = 0, (3.5) 
Q-C T HC-G = 0. (3.6) 

Let be the matrix satisfying Condition 13.11 Consider the hermitian form T : C n x C m 1— > R 
given by 

T(x,u) = x*Qx + 2Rex*7« + u*Yu. (3.7) 

Let g : C 1— > C nxra be the matrix- valued function 

5 (z) = (zI-A)- 1 , (3.8) 

We denote the unit circle by Q = {z G C : \z\ = 1}. 

The following Theorem establishes necessary and sufficient conditions for the existence of 
optimal u° for the problem (|2.ip - (|2.4|) 

Theorem 3.1. If there exists exists a u° G U such that $ (u°) < $ (u), for all u G U then 
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i) it is necessary that 

J r (g(z)bu,u) > 0, (Vz E C,Vn E C m ) . (3.9) 

ii j Furthermore, if there exists a 5 > suc/t i/iai 

F{g{z)bu,u) > S\u\l, (Vz G (,Vn G C m ) , (3.10) 
t/ien ii° is unique (up to equivalence). 

Theorem 13.11 above is an analog of KS Lemma for discrete-time optimal stochastic control 
problem ()2. lj) - f)2.4[) . This is a discrete time version of a continuous-time result obtained in [6] 
for the case when 7 = and in Chapter 5 of |16] ) for the general 7. 

3.1 Proof of Theorem I3TT1 

Lemma 3.1. If ut E U, then sup t>0 E \xt\ 2 < +00 for the solution of system (j2.3|) - (|2.4[) . 
Proof. Let 

/it = Ex t , (3.11) 
M t = Bx t xJ (3.12) 

Prom d23|)- d23|) and (pUT]) . we have 

fH+i = Am + but t = 0,1,2,..., (3.13) 
Ho = Efl. (3.14) 

n n 

Note that [E a| = Yl (Eai) < ^ Eaf = E | a| < +00. Thus, using the fact that u t E U and 

i=l i =l 

p{A) < 1, it follows from ([3TT3]l - (l3TT4"] l that ||^|| < +00. 

From (gSD-dSaj) and (f3TT2l) . we have 

M t+ i = AM t A T + Ap t uJb T + bu t pjA T + bu t uJb T + CM t C T , (3.15) 
M = Eaa T . (3.16) 

Let Qt = AntuJb T + butfjj A T + butuJb T . Let us denote the j-th colum of a matrix D by 
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We define the vectors qt,mt £ C n as 



qt 







M« 






Mi 2 ) 






Mi n) 



(3.17) 



The vectors qt and m( are formed by stacking up the columns of the matrices Qt and Mt, 
respectively. Set A=A®A+C®C (where ® denotes the Kronecker product). We can then 
rewrite (|3,15p as 



mt+i 



Am t + q t - 



(3.18) 



Note that the system in ([3.18p is of dimension n 2 , however, due symmetry, it can be reduced to 



a system of dimension 



n + n 



The assumption that the system (|2.3p - (|2.4p is stable in the mean-square sense for m = 0, 
is equivalent to mt being stable for qt = 0, which is true if and only if the spectral radius of A 
is p(A) < 1. From the solution of (|3.18p . we can show, using Holder's inequality and Young's 

I 1 2 

theorem, that Hmt^ < +oo, therefore sup t>0 E \xt\ < +oo. This compoletes the proof of 
Lemma 13. II □ 



It follows from Lemma 13.11 that the Z-transform, x(z), of Xf, exists, and it's radius of 
convergence contains the unit circle, £. If we set x% = 0, ut = for all t < we can then take 
the Z-transform of the system (|2.3p - ()2.4p and obtain 



X (Z) 



zg(z)a + g(z)bu(z) + g(z)C ^ 6+1 ^ 



Let D be an n x n real symmetric matrix and let T i— > 

T(D) = -L<fc T g(z) J Dg(z)C-dz. 
Lemma 3.2. Condition \3.1\ is satisfied if and only if Q satisfies 

g = e-r(e). 

Proof. Suppose there exists a such that (|3.2ip holds. Let 

H = -L(fg( Z ) r eg( Z )ldz. 
l-Kl Z 



Z" 

t= — QO 

nxn x R nxn bg defined by 



(3.19) 



(3.20) 



(3.21) 



(3.22) 



It follows from Parseval's identity that H = Yl (A 1 ) 1 @A l = 9 + A T HA. It therefore follows 

t=o 

from ([3^T]) that G = 9 - C T HC. Hence Condition O is satisfied. 

+00 , +00 . , , + OO , 

Conversely, suppose ([33]) holds, then £ (A ) i/A* — £ (yl T ) + = £ (,4 T ) 9,4*. 

It follows from Parseval's Identity that H = - — : <j> g (z) T Qg(z)—dz and it follows from (|3.6p 
and ([3T2T1 that G = 9-T(9). Thus ([3^T]) holds. This completes the proof of Lemma E3 □ 



Lemma 3.3. If the system (I2.3p - (|2.4p is stable in the mean-square sense for ut = £/ien 
Condition\3.1\ holds. 



Proof. Let us denote the i-th column of a matrix Z) by D_ j, let the matrices and G be as in 
Condition [3J] and let h = [flj, . . . , fljj , = [Gj 1; . . . , 9., n T ] and 5 = [G^, . . . , Gj n ] . Let 
A\ = A T <S> A T — I n 2 and A2 = —C T <X> G T , where I n 2 is the n 2 x n 2 identity matrix. We can 
rewrite (|3.5|) - (|3.6p as 



(3.23) 



Ax 


In 2 




h 







A 2 


4 2 
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Please notice that the system in f|3.23|) would be degenerate if and only if A\ = A2; however, 
this would require that A = A(g>A + C(&C = I n 2 which would violate the assumption that 
the matrix A from (|3.18|) satisfies p (A) < 1 (which is equivalent to the requirement that the 
system (|2.3p - (|2.4p be stable in the mean-square sense for ut = 0). Therefore, if the mean-square 
stability is satisfied, we can assume that the system in (|3.23p always has a solution, [/i T ,(/ T ] T . 

□ 



Hence matrices H and G exist (that is, Condition 13.11 holds). This proves Lemma 13.31 

It follows from Lemma 13.21 that G = 9 — T (9). Therefore, if we set xt = and ut = for 
t < 0, we can rewrite (|2.ip as 

+00 +00 



$(u) = E^(x t ,ut)- Bx* t T(Q)x t 

t=— 00 t=— 00 

Let the matrix-valued function n : C 1— > C mxm be defined by 

n (z) = b T g (z) T &g (z) b + b T g (z) T 7 + 7 T 5 (z) b + T, 



(3.24) 



(3.25) 
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and let 

(u(z),Ru(z)) = — I u(z)*U(z)u(z)-dz, (3.26) 
2m' z 

(r,u(z)) = (fEza T g(z) T [Gg(z)b + 1 ]u(z)-dz, (3.27) 

2m J(- z 

p = — (bBa 1 'g(z) J Gg(z)a-dz. (3.28) 
2m z 

It follows from (|3.26[) - (|3.28|) and Parseval's identity that we can rewrite (|3.24p as 

$(u) = (u(z) ,Ru(z)) + (r,u(z)) + p. (3.29) 

Thus. <3? (u.) is a quadratic form in u (.). Consider the deterministic control problem below. 
Minimize 

+oo 

= ^2[y*t&x t + 2Reyl-fu t + u* t r Ut } (3.30) 



over the set 



U = I u t € R m : \ u t\ 2 < +°° \ ( 3 - 31 ) 



i. t=o ; 

subject to 

Ite+i = Ajft + ftui, 4 = 0,1,2,... (3.32) 
y = Eo. (3.33) 

Here ?/i is an n-vector of states and itt is an m-vector of controls. Let matrices G, 7, T, A, and b 
and the vector a have the same properties as in the stochastic optimization problem (|2.1|) - (|2.4p 
above, and let the matrix be such that (|3.2ip is satisfied. Using Parseval's identity, we can 
rewrite (|3.30p as $1 (u (.)) = (u (.) , R\u (.)) + (r%, u (.)) + p\, where 

(u(z),R 1 u(z)) = — (pu(z)*Ii(z)u(z)-dz, (3.34) 
2m z 

(r u u(z)) = -L <fEza T g(z) T [Gg (z) b + 7 ]u (z) -dz, (3.35) 
2m z 

pi = — J)F,a T g(z) T Gg(z)a-dz. (3.36) 
2m z 

Theorem 3.2. An optimal control u° for the stochastic optimization problem (|2.ip - (|2.4p exists 
if and only if an optimal control for the deterministic optimization problem (|3.30p - (|3.33p exists. 
Furthermore, if (|3.10|) holds then the optimal controls in optimization problems (j2.1|) - (|2.4|) and 
(I3.30p - (|3.33p are identical and unique to within equivalence. 



Proof. Note that, the necessary and sufficient conditions for the existence of optimal u° that 
minimizes the quadratic form (u, Ru) + (r, u) + p depend on R and r. Moreover, the optimal 
u°, when it exists, is given by the solution to Ru° + r = 0. We can see from (|3.26p - (|3.28p and 
(|3,34p - (|3.36|) that R\ = R and r\ = r for the functionals <& and <3?i. It therefore follows that the 
solution to ()2.ip - (|2.4p exists if and only if the solution to (|3.30|) - ()3.33[) exists. Furthermore, if 
(|3,10p holds, it follows from the results from [25j[26], that the optimal control for (|3.30p - (|3,33p 
exists and is unique. This completes the proof of Theorem 13.21 □ 

It follows that if the optimization problem ()3.30p - ()3.33p has an optimal solution then (|3.9p 
must hold. Furthermore if (|3.10p holds, it follows that the solution exists and is unique (up to 
to within equivalence) . Hence the proof for Theorem 13.11 follows from Theorem 13.21 

Remark 3.1. If C = in the problem (|2.ip - (|2.4p . then the requirement that the system fj2.3j) - 
(|2.4|) be stable in the mean-square sense for Ut = will be equivalent to requiring that the matrix 
A satisfy p(A) < 1. In addition, if we set Q = G then (|3,2ip holds. Therefore Condition 6.1 
is satisfied and J- (x,u) = x*Gx + 2~Rex*jx + u*Tu, and Theorem \3.1\ will be the same as the 
results from [25, 26] with xo = Ea. 

3.2 Numerical Algorithm 

In this section we provide a Matlab code that takes matrices G, A and C as inputs, then checks 
if the system is stable in the mean-square sense. If it is stable, the program calculates matrices 
9 and H. 

I 

'/FILE NAME: numerics. m 

"^DESCRIPTION : Check if the discrete-time Linear Quadratic Control 
°/ Problem is Solvable. That is, we calculate H and Theta 
% that satisfy: {A ; HA-H+Theta=0 , Theta-C'HC-G=0> 
'/.INPUTS: Matrices G, A, C 
"/.OUTPUT : Matirx Theta, H 

I 

function [Theta, H] = numericsl(G, A, C) 

/.Verify that the inputs are all square matrices of the same dimension 

sl=size(G); s2=size(A) ; s3=size(C); 

if ((sl(l)~=s2(l)) | (sl(l)~=s3(l)) | (s2(l)~=s3(D) I . . . 
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(sl(2)~=s2(2)) | (sl(2)~=s3(2)) | (s2 (2) ~=s3 (2) ) | . . . 
(sl(l)~=sl(2)) | (s2(l)~=s2(2)) | (s3(l)~=s3(2))) 
disp( 'ERROR! Dimension Mismatch'); 
return; 
end; 

y t 

°/ Get the symmetric part of G 
G=0.5*(G+G') ; 

% 

ZVerify that the spectral radius of A is less than 1 
if (max(abs(eig(A))) >= 1) 

disp( 'Matrix A is not convergent'); 

return; 
end; 

% 

ZVerify that the system is Exponential Bounded in the mean-square sense 
Big_A = kron(A,A)+kron(C,C) ; 
if (max(abs(eig(Big_A))) >= 1) 

disp('The system is not EMS stable'); 

return; 
end; 

% 

°/ Solve the system, i.e. Calculate matrices H and Theta 

Al=kron(A' ,A' )-eye(size(A' ) ."2) ; 

Bl=eye(size(A') . ~2) ; 

A2=-kron(C ,C) ; 

B2=eye(size(A') . "2) ; 

M=[A1, Bl; A2, B2] ; 

v= [zeros (size (G (:))); G (:)] ; 

solution=M\v; 

theta=solution(length(solution) /2+1 : length(solution) ) ; 
h=solution(l : length(solution) /2) ; 

% 
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"/Return outputs H and Theta and Terminate the program 
Theta=reshape(theta,size(A) ) ; 
H=reshape (h, size (A) ) ; 
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